Survey insight reporting system and method

ABSTRACT

In one embodiment, the invention can be a method of displaying survey results. The method can include, for each of a plurality of survey items, receiving survey responses from survey respondents, wherein each response is chosen from response options, the response options corresponding to an ordinal scale; each respondent is associated with a collection of respondent segments; and each response to each item by each survey respondent corresponds with a response score. The method can further include, for each item and respondent segment, determining a composite score based on the response score. Further, response groupings can be displayed according to a noteworthiness ranking that is based on at least two of the composite score, a change in the composite score, and a sum of weights for each respondent segment.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/943,990, filed Apr. 3, 2018, now U.S. Pat. No. 10,055,701,which is a continuation-in-part of U.S. patent application Ser. No.14/932,017, filed Nov. 4, 2015, which claims the benefit of U.S.Provisional Patent Application No. 62/075,097 filed Nov. 4, 2014. U.S.patent application Ser. No. 15/943,990 further claims the benefit ofU.S. Provisional Patent Application No. 62/480,684, filed on Apr. 3,2017. The disclosures of these references are incorporated herein byreference in their entireties.

BACKGROUND

Surveys can provide several benefits, including enabling a company tobetter understand its workforce. Survey items can address issues such asengagement, organizational health, and satisfaction. Employee feedbackcan help an employer diagnose problems and find new opportunities forimprovement.

When responses to a survey correspond to an ordinal scale, there can bechallenges in accurately summarizing those responses. Responsesaccording to an ordinal scale are responses that can be ranked in anorder and thereby sorted. Likert scale responses (e.g.,Strongly-Disagree/Disagree/Neutral/Agree/Strongly-Agree, orBad/Needs-Improvement/Good/Excellent) are an example of ordinal scaleresponses.

There are two common methods for summarizing groups of responses to anordinal scale so that the groups of responses can be compared. In thepercent positive approach, scale options are divided into “positive” and“not positive” groups. The summary of a group of responses is thepercentage of the responses that are positive.

Another method is the integer assignment approach. In this approach,increasing successive integers are assigned to each of the scaleoptions. The summary of a group of responses is then the arithmeticmean, geometric mean, or median of the assigned integers.

Further, there are many methods for comparing a group score (e.g., ascore for a company) to a benchmark score. In the subtraction method,the benchmark score is subtracted from the group score, and the result,which can be positive or negative, is the distance from the benchmark.For example, a group might be 5 percentage points more positive than thebenchmark. In the ranks method, if the benchmark data can be dividedinto groups that are similar to the group being compared to thebenchmark, then the groups can be ordered and a rank for the groupcalculated. For example, a group might be 3^(rd) out of 30. Thepercentile method is the same as the ranks method, but instead ofreporting 3^(rd) out of 30, it is reported as the 90^(th) percentile.The z-score method is built on dividing the benchmark into groupssimilar to the group being compared to the benchmark. In this method,the standard deviation for the set of benchmark scores is calculated.Then, the subtraction method described above is used. But instead ofreporting the difference, the difference is divided by the benchmark'sstandard deviation resulting in a number known as a z-score, which isessentially the number of standard deviations the group score is fromthe benchmark average.

Percent Positive Approach

There are shortcomings with the above approaches. For example, with thepercent positive approach, the positive response options (agree,strongly agree) can be grouped together and the score reported can bethe percentage of all responders who responded positively. Thesepercentages can then be compared to percentages from similar companies.For example, a company may receive a score of 85% positive on an itemabout appreciation. The fact that a group of similar companies averaged73% positive on the appreciation item would lead the company to concludethat they were doing well at appreciating their employees.

The fundamental drawback to this approach is information loss. Bycollapsing multiple response options (typically five or seven) into two(positive or not positive), information is lost. Specifically,enthusiasm level is lost. No distinction is made between a positiveperson and an enthusiastic person. Similarly, no distinction is madebetween a neutral person and an angry person. Information loss can beacceptable when there is no benchmark data available or only poorbenchmark data available. With good benchmark data available, however,the loss of information can interfere with making better decisions abouthow to take cost-effective action.

Integer Assignment Approach

As another example, the integer assignment approach discussed above alsohas shortcomings. Table 1 below shows an example mapping of aseven-category Likert-type scale to the integers zero through six.Scores are then reported by converting each category on the responsescale to an integer using the mapping and then calculating thearithmetic mean of the integers.

TABLE 1 Category Integer Strongly disagree 0 Disagree 1 Slightlydisagree 2 Neutral 3 Slightly agree 4 Agree 5 Strongly agree 6

These averages can then be compared to averages from similar companies.For example, a company might get a score of 4.38 on an item aboutappreciation. The fact that a group of similar companies averaged 3.97on the appreciation item would lead the company to conclude that theywere doing well at appreciating their employees.

The integer assignment approach (sometimes referred to as the averagescore approach) improves on the percent positive approach in that eachresponse option is treated differently, and thus information is notlost. But this approach suffers from the same lack of calibrationdrawback as the percent positive approach, and it introduces the newdrawback of assuming that the response options are all equidistant fromeach other.

Every survey item has its own unique response characteristics. The mostobvious characteristic that varies by item is how easy the item is toagree with. For example, across a broad population, an item about paywill always receive a lower average than an item about ethics. This isbecause employees generally have a more negative view of their pay thanthey do of their companies' ethics. These differences can be adjustedfor by comparing the average score to a benchmark value.

In addition to different expected values, however, each item also has adifferent distribution or spread of responses. Put another way,different items have different degrees of expected variation in theresponses. Because of this, it can be difficult to compare an averagescore to a benchmark and know whether the difference is large orinconsequential. The same difference can be large given the distributionof one survey item, and inconsequential given the distribution ofanother survey item.

For example, consider two survey items: an item about whether anemployee thinks the company operates by strong ethics (ethics) and anitem about how much negativity there is at their company (negativity).The negativity item will typically elicit more enthusiasm in bothdirections with far more people responding both strongly negative andstrongly positive. Thus, the same size difference from the benchmark isfar more important for the ethics item than it is for the negativityitem. Put another way, a strongly negative response to the ethics itemis rarer and more worrying than a strongly negative response to thenegativity item.

Lack of calibration has varying levels of impact based on what kinds ofscores are being looked at. Most obviously, this drawback is a bigconcern when looking at a single population and comparing how theyscored on different items. Less obviously, this drawback is also aconcern when comparing how different populations, like departments orlocations, scored on a set of items. The issue here is that a problem ona narrowly distributed item can be masked by superficially good scoreson items with wider distributions.

To understand this drawback, it's important to understand the differentvariable types. There are four kinds of variables: categorical (ornominal) variables, ordinal variables, interval variables, and ratiovariables. A categorical variable is a variable in which the potentialvalues do not have any intrinsic order. An example categorical variableis eye color. It is possible to say that one eye color is equal or notequal to another eye color, but it is not possible to say whether oneeye color is greater than or less than another eye color.

An ordinal variable is a variable in which the potential values for thevariable do have a clear agreed upon ordering, but no clear consensus onthe relative distances between the values. An example of an ordinalvariable is the job grades of team member, manager, and senior manager.It can be agreed that in terms of the hierarchy of the organization, ateam member is at the lowest level, a senior manager is at the highestlevel, and a manager is in between. It's impossible to say, however,whether the gap between team member and manager is larger or the gapbetween manager and senior manager is larger, and even if one could, itwould be impossible to say by how much. One characteristic of ordinalvariables is that adding them or taking the arithmetic mean does notmake sense. A team member cannot be “added to” a senior manager to getthe result of two managers.

An interval variable is a variable where the potential values have botha clear agreed upon ordering and clear agreement about the distancebetween the values. An example of an interval variable is number ofhours worked per week. In this case, adding or taking the arithmeticmean of two variable values makes sense. If one person worked 20 hoursin a week and another person worked 40 hours in a week, then we couldreasonably say that the two people worked a total of 60 hours. Or, iftogether they made up a small department, we could say that on average,people in that department work 30 hours per week.

A ratio variable is a variable that includes all the characteristics ofan interval variable, with the additional condition that it includes ameaningful zero, in which zero indicates that there is none of thatvariable. For example, number of hours worked per week has a clear orderand distance between values, as with an interval variable. It is a ratiovariable because a value of zero indicates that the individual did notwork any hours that week.

A response on a Likert-type scale to a survey item is an ordinalvariable. For example, it is clear that being strongly negative is worsethan being negative, which is worse than being slightly negative. But aresponse on a Likert-type scale to a survey item is not an intervalvariable. For example, there is no definitive way to say that thedistance between strongly disagree and disagree is the same as thedistance from disagree to slightly disagree. Or, applying the arithmeticmean test, if you have a two-person department with one person whostrongly agrees with a statement and one person who strongly disagreeswith the statement, it's nonsensical to say that on average you haveneutral people. On some level, having two people who passionatelydisagree with one another is exactly the opposite of having two neutralpeople.

The integer assignment approach, however, forces inherently ordinalresponses on a Likert-type scale into being an interval variable. Whilethis approach is simple, it is mathematically unfounded and introduceserror into the scores. The equidistant point assumption drawback can beacceptable when looking at changes in scores from one time period to thenext across larger populations in situations where no external benchmarkdata or only poor external benchmark data is available. In these cases,the noise will often balance out, and avoiding the information lossinherent in the percent positive approach is valuable. But in othercases, this approach can lead to unreliable results.

Z-Score Approach

The z-score approach is the least common and most sophisticated of thethree common approaches for calculating survey scores. It can start withthe same calculation used in the integer assignment approach. Next, thedistance between the company score and the benchmark score is calculatedby subtraction, which is also the same as the integer assignmentapproach. But in a third step, instead of reporting the calculateddifference, the standard deviation of the scores that went into thebenchmark average is calculated and the difference between the companyscore and benchmark score is divided by the standard deviation.

This approach improves on the average score approach by providing somecalibration for the different response distributions of different surveyitems. It's easier to tell whether a particular score is an outlier ornot because this approach reports smaller differences on items withtighter distributions as bigger, and it reports bigger differences onitems with wide distributions as smaller.

At the same time, however, the z-score approach builds on the integerassignment approach by incorporating and amplifying the drawbacks of theinteger assignment approach. Because calculating integer-based averagesis the first step of the approach, the same equidistant point assumptiondrawback described under the subsection about the average scoresapproach also applies to the z-score approach. This z-score approachgoes on to make this drawback worse because, just like the averagecalculation, the standard deviation calculation assumes interval datarather than the merely ordinal data generated by surveys using aLikert-type scale. The standard deviation calculation also assumes anormal distribution. However, the responses to most survey items thatuse a Likert-type scale are skewed towards one of the ends of the set ofresponse options. In rare cases, responses are polarized with a dip inthe middle. These issues can make the results of the standard deviationcalculation misleading.

As discussed above, there are several shortcomings to the above methodsfor summarizing survey responses. These shortcomings interfere with theability to systematically identify noteworthy insights hiding in surveyresponse data. Previously, seasoned survey experts have had tosubjectively and often inaccurately adjust for these shortcomings whenmaking recommendations to organizations on what to focus on orcelebrate. What is needed is an improved system and method for receivingsurvey responses, more accurately summarizing these responses andsegments of these responses, and more accurately ranking the noteworthyaspects of these responses.

BRIEF SUMMARY

In one embodiment, a method of displaying survey results is disclosed,the method comprising: for each of a plurality of survey items,receiving survey responses from survey respondents, wherein: eachresponse is chosen from response options, the response optionscorresponding to an ordinal scale; and each respondent is associatedwith a collection of respondent segments; and for each response to eachitem by each survey respondent, determining a calibrated score, thecalibrated score based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide either: the response to the item or any of the responseoptions to the item that are lower on the ordinal scale; or the responseto the item or any of the response options to the item that are higheron the ordinal scale; for each item, determining a composite score basedon the calibrated score for each response to the item; for eachrespondent segment, determining a composite score based on thecalibrated score for each response associated with the respondentsegment; and providing a display of response groupings, the displayedresponse groupings comprising at least a portion of the plurality ofitems and the collection of respondent segments; wherein the displayedresponse groupings are located on the display along a first axis; andwherein the display includes, for each response grouping, acorresponding image, the corresponding image being located with respectto the first axis or a distinct second axis based on the composite scorefor the response grouping.

In another embodiment, a method of displaying survey results isdisclosed, the method comprising: for each of a plurality of surveyitems, receiving survey responses from survey respondents, wherein: eachresponse is chosen from response options, the response optionscorresponding to an ordinal scale; and each respondent is associatedwith a collection of respondent segments; and for each response to eachitem by each survey respondent, determining a calibrated score, thecalibrated score based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide either: the response to the item or any of the responseoptions to the item that are lower on the ordinal scale; or the responseto the item or any of the response options to the item that are higheron the ordinal scale; determining at least one of: for each item, acomposite score based on the calibrated score for each response to theitem; and for each respondent segment, a composite score based on thecalibrated score for each response associated with the respondentsegment; and providing a display of images, each image correspondingwith one of the item or respondent segments; wherein each image islocated with respect to a first axis based on the composite scoreassociated with the image and its corresponding item or segment.

In another embodiment, a method of displaying survey results isdisclosed, the method comprising: for each of a plurality of surveyitems, receiving survey responses from survey respondents, wherein: eachresponse is chosen from response options, the response optionscorresponding to an ordinal scale; and each respondent is associatedwith a collection of respondent segments; and for each response to eachitem by each survey respondent, determining a calibrated score, thecalibrated score based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide either: the response to the item or any of the responseoptions to the item that are lower on the ordinal scale; or the responseto the item or any of the response options to the item that are higheron the ordinal scale; determining an organization composite score basedon the calibrated score for each response; and providing a displaycomprising images corresponding with the organization composite scoreand other organization composite scores; wherein each image is locatedwith respect to a first axis based on the associated organizationcomposite score.

In another embodiment, a system for displaying survey results isdisclosed, the system comprising: respondent devices configured toreceive survey items and communicate survey responses to the items; anda server configured to carry out the steps of: for each item, receivingsurvey responses from respondent devices, wherein: each response ischosen from response options, the response options corresponding to anordinal scale; and each respondent is associated with a collection ofrespondent segments; and for each response to each item by each surveyrespondent, determining a calibrated score, the calibrated score basedon a probability that a person having the collection of respondentsegments associated with the survey respondent would provide either: theresponse to the item or any of the response options to the item that arelower on the ordinal scale; or the response to the item or any of theresponse options to the item that are higher on the ordinal scale; foreach item, determining a composite score based on the calibrated scorefor each response to the item; for each respondent segment, determininga composite score based on the calibrated score for each responseassociated with the respondent segment; and providing a display ofresponse groupings, the displayed response groupings comprising at leasta portion of the plurality of items and the collection of respondentsegments; wherein the response groupings are located on the displayalong a first axis; and wherein the display includes, for each responsegrouping, a corresponding image, the corresponding image being locatedwith respect to the first axis or a distinct second axis based on thecomposite score for the response grouping.

In another embodiment, a non-transitory computer-readable storage mediumencoded with instructions is disclosed, which, when executed on aprocessor, perform the method of: for each of a plurality of surveyitems, receiving survey responses from survey respondents, wherein: eachresponse is chosen from response options, the response optionscorresponding to an ordinal scale; and each respondent is associatedwith a collection of respondent segments; for each response to each itemby each survey respondent, determining a calibrated score, thecalibrated score based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide either: the response to the item or any of the responseoptions to the item that are lower on the ordinal scale; or the responseto the item or any of the response options to the item that are higheron the ordinal scale; for each item, determining a composite score basedon the calibrated score for each response to the item; for eachrespondent segment, determining a composite score based on thecalibrated score for each response associated with the respondentsegment; and providing a display of response groupings, the displayedresponse groupings comprising at least a portion of the plurality ofitems and the collection of respondent segments; wherein the responsegroupings are located on the display along a first axis; and wherein thedisplay includes, for each response grouping, a corresponding image, thecorresponding image being located with respect to the first axis or adistinct second axis based on the composite score for the responsegrouping.

In another embodiment, a method of displaying survey results isdisclosed, the method comprising: for each of a plurality of surveyitems, receiving survey responses from survey respondents, wherein: eachresponse is chosen from response options, the response optionscorresponding to an ordinal scale; each respondent is associated with acollection of respondent segments; and each response to each item byeach survey respondent corresponds with a response score; for each item,determining a composite score based on the response score for eachresponse to the item; for each respondent segment, determining acomposite score based on the response score for each response associatedwith the respondent segment; and providing a display of responsegroupings, the displayed response groupings comprising at least aportion of the plurality of items and the collection of respondentsegments; wherein the response groupings are located on the displayalong a first axis; and wherein the display includes, for each responsegrouping, a corresponding image, the corresponding image being locatedwith respect to the first axis or a distinct second axis according to anoteworthiness ranking, the noteworthiness ranking being based on atleast two of: the composite score for the response grouping; a change inthe composite score for the response grouping from a previous compositescore for the response grouping; and a sum of weights for eachrespondent segment, where each survey respondent or possible surveyrespondent is assigned a weight.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings, wherein:

FIG. 1 is a system for communicating survey items and summarizing surveyresponses according to an embodiment.

FIG. 2 is a user interface for a respondent device according to anembodiment.

FIG. 3 is a display of a summary of survey responses according to anembodiment.

FIG. 4 is a graph showing the expected probability distribution forsurvey responses according to one embodiment.

FIG. 5 is a graph showing polarization in survey responses according toone embodiment.

FIG. 6 is a graph showing coalescence in survey responses according toone embodiment.

FIG. 7 is flowchart for a method for generating a display summarizingsurvey responses according to an embodiment.

FIG. 8 is a display of a summary of survey responses according toanother embodiment.

FIGS. 9A and 9B are Venn diagrams representing overlapping segmentsaccording to one embodiment.

FIGS. 10 and 11 are displays of timeline summaries of survey responsesaccording to an embodiment.

DETAILED DESCRIPTION

The following description of the preferred embodiment(s) is merelyexemplary in nature and is in no way intended to limit the invention orinventions. The description of illustrative embodiments is intended tobe read in connection with the accompanying drawings, which are to beconsidered part of the entire written description. In the description ofthe exemplary embodiments disclosed herein, any reference to directionor orientation is merely intended for convenience of description and isnot intended in any way to limit the scope of the present invention. Thediscussion herein describes and illustrates some possible non-limitingcombinations of features that may exist alone or in other combinationsof features. Furthermore, as used herein, the term “or” is to beinterpreted as a logical operator that results in true whenever one ormore of its operands are true. Furthermore, as used herein, the phrase“based on” is to be interpreted as meaning “based at least in part on,”and therefore is not limited to an interpretation of “based entirelyon.”

Features of the present invention may be implemented in software,hardware, firmware, or combinations thereof. The computer programsdescribed herein are not limited to any particular embodiment, and maybe implemented in an operating system, application program, foregroundor background processes, driver, or any combination thereof. Thecomputer programs may be executed on a single computer or serverprocessor or multiple computer or server processors.

Processors described herein may be any central processing unit (CPU),microprocessor, micro-controller, computational, or programmable deviceor circuit configured for executing computer program instructions (e.g.,code). Various processors may be embodied in computer and/or serverhardware of any suitable type (e.g., desktop, laptop, notebook, tablets,cellular phones, etc.) and may include all the usual ancillarycomponents necessary to form a functional data processing deviceincluding without limitation a bus, software and data storage such asvolatile and non-volatile memory, input/output devices, graphical userinterfaces (GUIs), removable data storage, and wired and/or wirelesscommunication interface devices including Wi-Fi, Bluetooth, LAN, etc.

Computer-executable instructions or programs (e.g., software or code)and data described herein may be programmed into and tangibly embodiedin a non-transitory computer-readable medium that is accessible to andretrievable by a respective processor as described herein whichconfigures and directs the processor to perform the desired functionsand processes by executing the instructions encoded in the medium. Adevice embodying a programmable processor configured to suchnon-transitory computer-executable instructions or programs may bereferred to as a “programmable device”, or “device”, and multipleprogrammable devices in mutual communication may be referred to as a“programmable system.” It should be noted that non-transitory“computer-readable medium” as described herein may include, withoutlimitation, any suitable volatile or non-volatile memory includingrandom access memory (RAM) and various types thereof, read-only memory(ROM) and various types thereof, USB flash memory, and magnetic oroptical data storage devices (e.g., internal/external hard disks, floppydiscs, magnetic tape CD-ROM, DVD-ROM, optical disk, ZIP™ drive, Blu-raydisk, and others), which may be written to and/or read by a processoroperably connected to the medium.

In certain embodiments, the present invention may be embodied in theform of computer-implemented processes and apparatuses such asprocessor-based data processing and communication systems or computersystems for practicing those processes. The present invention may alsobe embodied in the form of software or computer program code embodied ina non-transitory computer-readable storage medium, which when loadedinto and executed by the data processing and communications systems orcomputer systems, the computer program code segments configure theprocessor to create specific logic circuits configured for implementingthe processes.

Determining Probability of a Response Option

In the exemplified approach for summarizing ordinal scale responses,there is a determination of the probability of each response option orlower being chosen, either using benchmark data or using the responsesfrom the group itself. A method for determining such a probability isshown in the following example, where there are four scale options forresponse: “Bad,” “Needs Improvement,” “Good,” and “Excellent.” Onehundred responses were received, specifically, ten “Bad” responses,twenty “Needs Improvement” responses, forty “Good” responses, and thirty“Excellent” responses. These response results are shown in Table 2below.

TABLE 2 Needs Bad improvement Good Excellent Response count 10 20 40 30Percentage 10% 20% 40% 30% Percentile  5% 20% 50% 85% Odds 1:19 (0.053)1:4 (0.25) 1:1 (1) 17:3 (5.67) Log odds (logits) −2.94 −1.39 0 1.73Centilogits −294 −139 0 173

The percentages these responses represent for the total responses are,respectively, 10%, 20%, 40%, and 30%. Because the scale lacksgranularity, it is assumed that half the responses to a particularoption were intended to be more positive than was able to be expressed(and that half were intended to be less positive). Thus, the probabilityor percentile for each response option is calculated as half thepercentage for the response option, plus the percentages of the lowerresponse option. Thus, the Bad percentile is 10%/2 (or 5%), indicatingthere is a 5% probability that the response option chosen will be Bad.Similarly, the Needs Improvement percentile is 20%/2+10% (or 20%), thusindicating that there is a 20% chance that the response option chosenwill be Needs Improvement or Bad. Similarly, the Good percentile is40%/2+10%+20% (or 50%), and the Excellent percentile is30%/2+10%+20%+40% (or 85%). In other embodiments, other percentages(other than 50%) can be used to determine the percentile. Further, moreelaborate techniques can consider the “curve” of responses across theentire ordinal scale.

Next, from the percentiles (probabilities), the odds of a particularscore or lower occurring can be determined for each response option. Forsomething with a 5% probability, there is one “chance” that it willhappen and 19 “chances” that it won't. Thus, the Bad odds are 1 to 19 or0.053, the Needs Improvement odds are 1 to 4 or 0.25, the Good odds are1 to 1 or 1, and the Excellent odds are 17 to 3 or 5.67.

From the odds, a logit score can be calculated for each response option,the logit values ranging from negative infinity to positive infinity.The Bad logit score is ln( 1/19) (or −2.94), the Needs Improvement logitscore is ln(¼) (or −1.39), the Good logit score is ln(1/1) (or 0), andthe Excellent logit score is ln(17/3) (or 1.73). These scores can beexpressed in centilogits. A centilogit is one hundredth of a logit.Thus, as indicated in the table above, the respective scores incentilogits are −294, −139, 0, and 173.

These logit scores can be assigned to each response option. Accordingly,the logit scores can be used in a manner similar to the use of integersin the integer assignment approach (discussed above). Using logit scoresinstead of the integers helps to appropriately weight passion andoutliers, which typically leads to better group decision making.

Further, the logit scores can be used to generate a summary score for asurvey item. For example, suppose the above table represents the onehundred responses from five survey items asked of twenty differentpeople. A summary score for a given one of these survey items where tenresponders rated it Excellent, five rated it Good, four rated it NeedsImprovement, and one rated it Bad can be obtained by applying thecalculated logit scores to the response counts as follows:

${{Summary}\mspace{14mu} {score}} = {\frac{{( {- 294} )(1)} + {( {- 139} )(4)} + {(0)(5)} + {(173)(10)}}{20} = 44}$

The summary score can then be compared to the other four summary scoresfor the other survey items.

In the example provided above, the percentiles and logit scores aredetermined based on the survey responses received. While this approachcan weight passion and outliers, a more preferred approach is to obtainpercentiles and logit scores from benchmark data. These numbers can beobtained in the same manner as discussed above, but using benchmarksurvey responses from other companies. For example, using the aboveapproach, benchmark data from a similarly situated company (orcompanies) can be used to determine the percentiles and logit scores foreach response option for each survey item. The calculated logit scorescan then be applied to the responses of the company being surveyed in amanner similar to the integer assignment approach. When using logitscores obtained from benchmark data, the system can granularly adjustfor variations in item wording and demographic characteristics ofresponders.

Respondent Segments

In the exemplified embodiment, each survey respondent is associated witha collection of respondent segments. As used herein, the term “segment”or “respondent segment” can be any category, group, division, orclassification by which by respondents and/or potential respondents canbe grouped. A segment can be, for example, a demographic segment, suchas respondent tenure (e.g., 5-10 years tenure), respondent hours worked(e.g., full-time or part-time), respondent salary range (e.g., over$50,000), respondent management level (e.g., manager), respondentdepartment type (e.g., marketing), respondent role (e.g., paralegal),and country (e.g., India or Great Britain). The respondent segments canalso be understood to include the respondent identities themselves, suchas the name of an employee (e.g., John Smith) providing feedbackregarding the organization to which he belongs.

The above approach can take into consideration certainexpectation-setting segments of an organization. Expectation-settingsegments are segments for which there is an expectation for howrespondents belonging to the segment will respond to certain surveyitems. For example, if asked about the direction of the company, seniormanager responses are likely to be unbalanced towards the positive,since they are generally responsible for setting the direction of thecompany. Thus, positive responses by this segment should not be taken asseriously as moderate or low responses. By contrast, lower-levelemployees are likely to be unbalanced towards the negative in responseto this direction item, and thus positive responses from this segmentwould be more noteworthy. In an opposite manner, if asked about theexecution of the direction of the company, senior managers are likely tobe unbalanced towards the negative, and lower-level employees are likelyto be unbalanced towards the positive. Thus, respondent management levelcan be considered an expectation-setting segment. Further, one mightexpect that people of a certain country (e.g., India) will respond tosurveys more positively than people of another country (e.g., Europe),while people of other countries are expected to be in between. Otherexamples of potentially expectation-setting segments are respondenttenure, respondent salary range, and respondent department type. Byconsidering a respondent's collection of associated segments andconsidering, based on those segments, what response to a survey item isexpected, a survey system can better determine when responses are trulyout of the ordinary and more noteworthy.

By contrast, other segments are not expectation-setting segments. Suchsegments can include (though do not necessarily include) location (theremay be no expectation that employees in a Maine office will responddifferently from those in a Vermont office), company-unique departments,and a segment representing a single individual. Such segments can bedisregarded for purposes of setting expectations for a given surveyitem.

Calibrated Score

The above approach for determining a probability can be used todetermine a custom score (e.g., a logit score) for each survey responseto each survey item by each survey respondent. Each score can be basedon the collection of expectation-setting segments that are associatedwith the survey respondent. More specifically, the score can be based ona probability that a person having the collection of respondentexpectation-setting segments associated with the survey respondent wouldprovide the survey response to the item or any of the response optionsto the item that are lower on the ordinal scale. A score based on thisprobability is referred to herein as a “calibrated score.” In theexemplified embodiment, a logit score (in logits or centilogits) is usedfor the calibrated score. But the calibrated score is not so limited, asit can be another number based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide the survey response or lower to the item, including theprobability itself. For example, such a number could be a straightprobability or straight odds.

For example, the exemplified embodiment can use benchmark data (or asuperset of the responses being summarized) to calculate the probabilitythat a full-time manager who makes between $50,000 and $100,000 and washired between one and three years ago would respond to a specificstatement about the direction of their company with a response of“Slightly Agree.” As discussed above, the exemplified approach can usethe item response count from the benchmark data to determine thepercentage of benchmark respondents responding Slightly Agree or lower.This percentage is then divided in half to compensate for the ordinalscale's lack of full granularity, and to estimate the proportion of thatoption's responses that were actually intended to be more positive thanwas able to be expressed on the ordinal scale. This percentile can thenbe used, according to the method discussed above, to determine acalibrated score specific to the response (Slightly Agree), the surveyitem (regarding company direction), and the collection of segments towhich the respondent belongs. In the exemplified embodiment, thecalibrated score can be a logit-based score.

Survey System

Referring now to the figures, FIG. 1 is a system 10 for communicatingsurvey items and summarizing survey responses according to anembodiment. The exemplified system 10 includes a server 12 and router 16connected to the internet 14, as well as respondent devices 22. Therespondent devices 22 belong to respondents 20.

In the exemplified embodiment, the system can provide survey items to apotential respondent in real-time, and can receive survey responses inreal-time. For example, after a meeting, a manager can immediately senda survey item to the participants, and can view the responses in aresponse report that updates in real-time as the survey responses arereceived. In other embodiments, the method can use less rapidcommunications. In yet other embodiments, a more standard survey, suchas a paper survey, can be used.

As used herein, the term “survey item” or “item” can refer to anystatement, question, or topic for which a respondent can provide aresponse or rating. For example, an item can be the statement “I thinkthe company is headed in a positive direction,” and the response optionscan be Strongly Disagree, Disagree, Slightly Disagree, Neutral, SlightlyAgree, Agree, and Strongly Agree.

The server 12 can be any computer or processor (or collection thereof)for carrying out programs in accordance with the functions describedherein. In the exemplified embodiment, the server 12 communicates withthe respondent devices 22 through an internet connection, the router 16providing wireless internet connection to the respondent devices 22. Inother embodiments, the server 12 can communicate with the respondentdevices 22 through any standard communication means, including throughuse of a telecommunication network (e.g., 3G or 4G) or a wired internetconnection (e.g., wired Ethernet cables).

FIG. 2 is a user interface 41 for a respondent device 22 according to anembodiment. In the exemplified embodiment, the respondent device 22 is amobile smartphone. In other embodiments, the respondent device 22 can beany computer device capable of carrying out programs in accordance withthe functions described herein (including laptop computers, desktopcomputers, and tablets). As stated above, in other embodiments, a moretraditional survey system (e.g., a paper survey) can be used.

The user interface 41 of the exemplified respondent device 22 shows afirst instance of a survey application 40. In the exemplifiedembodiment, the survey application 40 is a smartphone application. Inother embodiments, the survey application 40 can be any program forcarrying out the functions described herein. The respondent device 22provides the respondent user interface 41. In the exemplifiedembodiment, the user interface 41 utilizes a touch screen provided bythe smartphone. In other embodiments, the user interface can be any userinterface capable of enabling a user to communicate with and carry outthe functions described herein, including an interface utilizing acomputer monitor, mouse, and/or keyboard.

The respondent user interface 41 shows a survey item 42, namely, “Howare we doing at achieving our purpose?” The respondent user interface 41provides response options 45 for a response. In the exemplifiedembodiment, there are four ordinal scale response options 45 to choosefrom (from left to right): a red circle (Bad), an amber circle (NeedsImprovement), a green circle (Good), and a purple star (Exceptional).(There is also a fifth, non-ordinal scale response, a cloud representingNo Answer.) The ordinal scale response options 45 represent differentratings of how the company is doing regarding achieving theorganization's purpose. In the exemplified embodiment, the red circlerepresents the Bad rating, which can be described as follows: “There aresignificant problems that need to be dealt with urgently.” The ambercircle represents the Needs Improvement rating, which can be describedas follows: “There are obvious and valuable improvements that can bemade.” The green circle represents the Good rating, which can bedescribed as follows: “Nothing's perfect, but we are doing welloverall.” Finally, the purple star represents the Exceptional rating,which can be described as follows: “We are doing better here atachieving our purpose than anywhere else I know of.” In the exemplifiedembodiment, the red option has been chosen as the response 44. In otherembodiments, other rating options having other meanings can beavailable.

In the exemplified embodiment, there is also a request for comment 46Ain which the respondent is asked to comment on why the chosen response44 was given. In this embodiment, the respondent can provide anunstructured written comment 46. In the exemplified embodiment, therespondent states the organization has “Poor leadership.” In otherembodiments, structured responses (e.g., a list of possibleexplanations) can be provided for selection and/or unstructuredresponses can be eliminated.

Composite Scores

For each survey item, a composite score can be determined based on thecalibrated score for each survey response to the item. For example, thecomposite score for an item can be an average of the calibrated scoresassociated with the item. In the preferred embodiment, the compositescore is an average of the calibrated scores associated with the item.The average can be any type of average, including an arithmetic mean, ageometric mean, or a median. In yet other embodiments, the compositescore can be another score for summarizing the relevant calibratedscores. The composite score for each item can be used for quantifyingthe noteworthiness or interestingness of the responses to a given item.

A composite score can also be determined for each respondent segment,the composite score based on the calibrated score for each surveyresponse for the respondent segment. As with composite scores for items,the composite score for a segment can be determined as an average of thecalibrated scores or by any other means of summarizing the relevantcalibrated scores. The composite score for each segment can be used forquantifying the noteworthiness or interestingness of responses fromdifferent segments.

Generally, the term “composite score” for a given response grouping(e.g., survey item or segment) can refer to any single numericrepresentation of the response scores for that response grouping. The“response score” upon which the composite score is based can be acalibrated score (discussed herein) or any other type of numericrepresentation of an ordinal scale response, such as a z-score, a scoreindicating whether the response is positive or negative, or an integerassigned to the ordinal scale response. Thus, in certain embodiments,survey results can be displayed and ranked for noteworthiness withoutusing calibrated scores. For example, the response score for eachresponse can be determined based on whether the response is positive ornegative, an integer assignment, or a z-score.

Display

FIG. 3 is a display 100 of a summary of survey responses according to anembodiment. The exemplified display 100 includes a first axis 112 and asecond axis 114. Along the first axis are several survey responsegroupings 104. As used herein, the term “response grouping” or“grouping” can refer to any survey item, survey segment, or combinationof survey item(s) and/or survey segment(s) that forms a basis forgrouping responses together. For example, a response grouping can be allthe responses to a given survey item, or all the responses from a givensegment, or a combination thereof. As shown in FIG. 3, the responsegroupings 104 comprise survey items 106 and survey segments 108. Eachresponse grouping 104 has a corresponding image 110 that is located withrespect to the second axis 114 based on the composite score for theresponse grouping 104. In the exemplified embodiment, the correspondingimage 110 is a circle, though in other embodiments any other image(including a dot or a different shape) could be used. Further, the sizeof the images can be altered to reflect certain details about theassociated item. For example, each displayed segment can have an imagewhose size is based on the total number of the survey respondents (orthe total number of possible survey respondents) that correspond withthe displayed segment. Further, each displayed survey item can have animage whose size is based on an item multiplier. The item multiplier canbe based on any of the factors discussed herein with respect tomultipliers.

The display also includes a change indicator 111 indicating a magnitudeof a change in the composite score for the response grouping 104 from aprevious composite score for the response grouping 104. In thisembodiment, the change indicator 111 is an arrow and a length of thearrow is indicative of the magnitude of the change. One of the responsegroupings includes a polarization indicator 116 indicating polarizationof the survey responses for the response grouping. Further, one of theresponse groupings includes a coalescence indicator 118 indicatingcoalescence in the survey responses for the response grouping.Polarization and coalescence will be discussed in greater detail below.It is further noted that the exemplified system can continue receivingnew survey responses and update the determined scores and displayaccordingly. The system can also receive new benchmark data and updatethe scores and display accordingly.

Timeline View

As shown in FIGS. 10 and 11, the display 100 can provide timelinesummaries of survey responses. Thus, the display can include, for eachresponse grouping, a plurality of corresponding images and relatedcomposite scores for different times when survey responses werereceived, each of the plurality of corresponding images being located(a) with respect to the first axis, based on the composite score for theresponse grouping; and (b) with respect to the second axis, based on thetime when the survey responses related to the corresponding image werereceived.

In FIG. 10, the display 100 includes response grouping images 110 thatrelate to survey responses to survey items or topics regardingorganizational health. The organization in question has taken the surveyfor three years. The display 100 tracks the composite scores for thesesurvey items for each year. The images 100 are located with respect tothe first axis 112 based on the composite score for the correspondingitem. The images 100 are located with respect to the second axis 114based on the time of the survey responses. As can be seen, for severalsurvey items, company performance dropped in 2016, but improved in 2017.The lower third of scores are considered low to a potentially noteworthydegree, the middle third of scores are considered as not requiringfocused attention, and the higher third of scores are considered so goodas to be potentially noteworthy. This can be represented along the firstaxis 112 with colors red, gray, and green, respectively, and/or one ormore bands can be colored on the chart. Further, the color of the image(circle) can correspond with the third of the graph in which the imageis located. Further, the lines connecting the images over time can becolored to represent decline (red), improvement (green), or nopotentially noteworthy change (gray).

FIG. 11 is similar to FIG. 10, but the display 110 includes responsegrouping images that relate to demographic segments, specifically, jobgrade. The display 100 tracks the composite scores for these job gradesfor each of three years. The images 100 are located with respect to thefirst axis 112 based on the composite score for the job grade. Theimages 100 are located with respect to the second axis 114 based on thetime of the survey responses. As can be seen, for example, while SeniorManager and Team Member scores have been improving recently, Managerscores have not.

The exemplified display 100 of FIG. 3 can be a user interface 102 thatallows the user to select how the survey responses are displayed.Specifically, in the exemplified embodiment, the user interface 102includes a display control interface 119 that enables the user tocontrol how the response groupings 104 are displayed. In thisembodiment, display control interface 119 provides several drop-downmenus to determine which response groupings will be displayed and inwhat manner. In other embodiments, other means can be used for providingthe user options for determining how to display the response groupings,such as a series of windows providing the user questions regarding thedesired features of the display.

A segment drop-down menu 120 enables the user to select the segments theuser wishes to view. In this figure, the “Show all” option has beenselected, and thus all segments and items are eligible to be shown onthe display. Other options that can be provided by the segment drop downmenu 120 include any type of segment, such as a department, a location,employees hired a certain range of time ago, employees whose salary fallwithin a particular range, part-time employees, employees with a commutewithin a certain range, employees with a certain performance grade, oremployees at a specific management level.

A topic drop-down menu 122 enables the user to select whether to viewresults from all of the survey items or narrow down to a subset of itemsor even a single item. A grouping-type drop down menu 124 enables theuser to select how the results will be grouped, or put another way, whatresponse groupings will be shown. Because of the calibration provided bythe calibrated scores, all grouping types can be combined in one chart,but in other instances a more traditional display can be provided, suchas one just showing departments, just showing survey items, or justshowing a particular demographic.

A scores drop down menu 126 enables the user to select which scores aredisplayed. In this figure, the “Top and bottom scores only” option hasbeen selected, and thus the display is limited to showing only thehighest and lowest scores, which are likely to indicate the promisingand troubling issues for an organization. Other options that can beprovided by the scores drop down menu 126 include “Top scores only,”“Bottom scores only,” and “All scores.”

Noteworthiness

The user interface 102 further provides options for ordering theresponse groupings on the display. In this embodiment, an ordering dropdown menu 128 enables the user to select from one of several parameters130 according to which the displayed response groupings are ordered.Each of these parameters or options can be understood as a differentbasis for determining the noteworthiness, noteworthiness score, ornoteworthiness ranking of a response grouping. The noteworthiness of aresponse grouping is an indication of how worthy of attention the surveyresults for the response grouping are, and the noteworthiness score is anumerical representation of the noteworthiness. The noteworthinessranking is a ranking of a response grouping based on the noteworthinessor noteworthiness score. Using menu 128, the user can select the basisfor determining noteworthiness that is most suitable for the particularorganization.

The ordering drop down menu 128 provides several parameters for orderingthe response groupings. The response groupings can be ordered simply bytheir composite score (the “Ordered by score” option). In this event,the noteworthiness of a particular response grouping is based on thecomposite score for the response grouping. As described above, for eachresponse grouping (whether an item or a segment of a combinationthereof), the composite score can be determined based on the relevantcalibrated scores.

The response groupings can also be ordered according to a score impact,which is a measure of the amount by which that response grouping ismoving the overall score of the selected segment from the segment dropdown menu 120 up or down. Put another way, if a response grouping wasremoved, how much would the selected segment's score be different?

The response groupings can also be ordered according to a change in thescore from a previous score for the same response grouping. Thus,noteworthiness can be based on a change in the composite score for aresponse grouping from a previous composite score for the responsegrouping. The response groupings can also be ordered according to achange impact, which is a measure of an amount by which that responsegrouping's change is affecting the overall change of the selectedsegment from the segment drop down menu 120. Put another way, if thatresponse grouping was removed, how much would the selected segment'schange score be different?

In certain cases, a previous composite score is unavailable, forexample, where there is a new department that does not have an analog ina prior department structure, or where a new item has been added to thesurvey. Indicating that these response groupings have no change from aprevious score is a poor estimation if there is noteworthy change at theorganization at large. In one embodiment, to address this issue for adepartment segment, an average change of a parent-department segment canbe used as a basis for determining noteworthiness. Further, otherrelated segments can be used where a previous score is unavailable for asegment.

The response groupings can also be ordered according to (and thusnoteworthiness based on) weight, which is a measure of how importanteach response grouping is. For segments, weight can be based on, forexample, a total number of the survey respondents associated with therespondent segment, or a total number of possible survey respondentsassociated with the respondent segment (respondent or not), an aggregatesalary for either of those groups (the bigger the salary, the moreweight is given to the response), an aggregate financial contributionmade by the group to the organization, some subjective strategicimportance multiplier, individual weights assigned to each member of asegment (which can be based on the foregoing), or some combinationthereof. Thus, for example, a department having a large number ofemployees can be given greater weight even if only a small number ofthose employees actually responded to the survey. The composite scorefor each department can be multiplied by the number of people in thedepartment. For survey items, weight can be based on, for example, anassigned multiplier, typically determined by correlating responses tothe item to some business metric of value to the organization.

The response groupings can also be ordered according to (and thusnoteworthiness based on) a combination of any of the metrics or otherfactors discussed herein, including the metrics for score and changediscussed above, and the overlap considerations discussed below. In FIG.3, noteworthiness is the selected option 132 for ordering the responsegroupings.

Noteworthiness can also or alternatively be based on other factors(including any combination of the factors discussed herein). Forexample, noteworthiness can be based on the details of a departmentsegment's parent-department segment. In one embodiment, the collectionof respondent segments comprises department segments, and the departmentsegments comprise a parent-department segment having child-departmentsegments. The noteworthiness of a child-department segment can be basedon a comparison of the composite scores for child-department segments.Departments that have scores that are very different from the scores oftheir peer departments under the same parent department can beconsidered more noteworthy.

Noteworthiness can also be based on coalescence or polarization in thesurvey responses for the response grouping. In a preferred embodiment,coalescence and polarization are determined at least in part bycomparing the survey responses for the response grouping to thebenchmark data relevant to the response grouping.

Coalescence and polarization can be calculated by first determining therange of probability of each scale option or lower being chosen for eachparticular combined segment responding to each particular survey itemusing benchmark data, which can either be external to the responsesbeing summarized or a superset of the responses being summarized. Eachresponse can then be mapped to that probability range. All responses areweighted equally, so responses are spread thinner across widerprobability ranges. If the responses perfectly match the benchmark, thenthe responses will be evenly spread across the full range ofprobability. However, this typically is not the case. Typically, theresponses will be unbalanced towards the positive or towards thenegative, reflecting an average score that is above or below average.However, sometimes the average score will be hiding that the responseswill be unbalanced towards both the positive and the negative at thesame time, and knowing this is very valuable for understanding how tobest interpret the average score and address the underlying issuesbehind it. To understand this, it's helpful to have a visualization ofexpectations and visualizations of coalescence and polarization.

In an alternative embodiment, the following sort orders are provided asoptions: sort by score (where composite score shows how the surveyresponses compare to what was expected from matching groups ofemployees), sort by change (where change compares this year's responsesto matching groupings of responses from last year), sort by company gap(where company gap is the difference between a department's score andthe entire company's score with the same demographic and statementfilters), sort by parent gap (where parent gap is the difference betweena department's score and its parent department's score with the samedemographic and statement filters), sort by importance (where importancecaptures the real-world impact of a particular score, specifically, hownoteworthily a statement or group influences engagement; for groups, itexpresses responder count; for statements, it expresses correlation withengagement (aka weight)), sort by blended (focuses on the mostinteresting results by sorting by a blend of score, change, importance,actionability, polarization, overlap awareness, actionability, and otherfactors (aka full noteworthiness)), sort by score and importance (sortby a blend of score and change (aka score impact)), sort by change andimportance (sort by a blend of change and importance (aka changeimpact)), sort by score and change (sort by a blend of score andchange), sort by score, change, and importance (sort by a blend ofscore, change, and importance), sort by importance and company gap (sortby a blend of importance and difference from overall company score),sort by importance and parent gap (sort by a blend of importance anddifference from parent), and sort by blended ignoring overlap (while theprimary “blended” sort deemphasizes lower priority overlapping groups,this sort ignores overlap). As noted elsewhere, any of these options orcombination of options discussed herein can be used for displaying andordering the response groupings.

Visualizing Expectations

FIG. 4 is a graph showing the expected probability distribution forsurvey responses according to one embodiment. As shown, the expectedprobability distribution can be visualized with a rectangle, with eachvertical slice representing a percentage of responses expected to bespread across that probability of having that score or more extreme.Where the expected probability distribution originated (in terms of theoriginal response scale) can be visualized by drawing curves thatrepresent how each response option (e.g., Strongly agree, Agree,Slightly Agree) is distributed within the rectangle representingexpectations. For example, curve 321 shows that all of the highestprobabilities (e.g., from about the 95% mark and up) came from StronglyAgree responses. From a probability mark of about 80% down to 95%, theresponses came from a combination of Strongly Agree and Agree (curve322). Below about 80%, a very small number of Slightly Agree responses(curve 323) started to be included. Around the 50% probability mark,there are no more Strongly Agree responses. Agree responses make up thebulk of the responses, and a small percentage of Slightly Agree andNeutral responses (curve 324) are contributing. The area under thedifferent colored curves varies and is proportional to the percentage ofresponses that came from the corresponding response options. Anothercharacteristic of the visualization is that if you add up the heights ofthe curves at any particular horizontal location, the sum will be theheight of the expectation rectangle.

If there are more responses in the probability extremes or “tails,” thisis indicative of polarization. FIG. 5 is a graph 300 showingpolarization in survey responses. The response options correspond withthe response scale of Strongly Disagree, Disagree, Slightly Disagree,Neutral, Slightly Agree, Agree, and Strongly Agree. The portion 302 isindicative of responses that are as expected. The negative tail 304 isindicative of the more negative responses than expected, and thepositive tail 306 is indicative of the more positive responses thanexpected.

By contrast, if there are more responses in the middle, this isindicative of coalescence. FIG. 6 is a graph 350 showing coalescence insurvey responses. Again, the response options correspond with theresponse scale of Strongly Disagree, Disagree, Slightly Disagree,Neutral, Slightly Agree, Agree, Strongly Agree. Portion 352 isindicative of responses that are as expected. Portion 356 is indicativeof unexpectedly high responses, and portion 354 is indicative ofunexpectedly low responses.

A sense of how unusual the distribution is can be calculated by lookingat how much the responses vary from the mean probability for thepopulation being summarized. One such technique is to calculate theaverage distance of the probability-spread responses from the meanprobability. Values for coalescence range from 0 (large) to 0.25 (none).Values for polarization range from 0.25 (none) to 0.5 (large).

Additional Noteworthiness Factors Such as Overlap

The noteworthiness of a response grouping can also be based on anactionability index that is indicative of the ease or difficulty inaddressing the response grouping. In one embodiment, actionability canbe based on the ease in getting a segment together. For example, it isharder to get the employees who have worked with the company from 1-3years together than it is to get the sales team together. Further, it isharder to get the sales team together than it is to get the Boston salesteam together. The actionability index can be used to identify responsegroupings that, when addressed, will have a higher impact per person.

Noteworthiness can be based on other or additional factors. For example,noteworthiness can take into consideration a composite score and thebiggest components of what is pulling it down (or up), includingoverlaps. Given a focus area, this allows the system to narrow downoptions and be smarter about matters such as demographics. For example,if Sales and the 10-15 Year tenure segment both have low scores, but itis the overlapping Sales members in the 10-15 Year tenure band that arepulling things down, the system can narrow in on that. Or if the systemdetermines that, for the most part, those groups completely overlap, thesystem can focus on Sales because that is an easier group to thinkabout.

In one embodiment, the method removes overlap where there is a propersubset. Specifically, the method comprises (a) among the respondentsegments, identifying a first segment and a second segment as havingoverlapping respondents; (b) identifying the first segment respondentsas being a proper subset of the second segment respondents; (c) reducingthe noteworthiness ranking of the second segment if a second segmentnoteworthiness score is closer to an average noteworthiness score than anoteworthiness score for survey respondents belonging to the secondsegment but not the first segment; and (d) reducing the noteworthinessranking of the first segment if a first segment noteworthiness score iscloser to an average noteworthiness score than a noteworthiness scorefor survey respondents belonging to the second segment but not the firstsegment. In other embodiments, step (c) or (d) can be omitted. Theaverage noteworthiness score can be based on any average ofnoteworthiness scores, including the average noteworthiness score forall the response groupings of an organization being surveyed, theaverage noteworthiness score for a relevant scope of response groupingswithin the organization, and an average noteworthiness score based onnoteworthiness scores from a collection of organizations.

For example, Sales in Boston can be considered a proper subset of SalesGlobal. FIG. 9A illustrates a proper subset, where Sales Global isrepresented by circle B and Sales Boston is represented by circle A,since all members of Sales Boston belong to Sales Global. Further,portion B-A represents those members of segment B that do not belong tosegment A (that is, those in Sales Global that do not belong to SalesBoston). In this example, the noteworthiness score of segment B iscompared with the noteworthiness score of segment B-A. If the segmentB-A score is closer to the average than the whole of segment B, thisindicates that subset segment A is the reason for the noteworthy (low orhigh) score, and therefore the system can treat the whole of segment Bas less noteworthy, and thus decrease segment B's noteworthinessranking. Otherwise, this indicates that segment B is the noteworthygroup to take action on, and the system can treat segment A as lessnoteworthy, and thus decrease segment A's noteworthiness ranking. In anembodiment, the difference between the scores mentioned above must be“wider” than a predetermined material amount (e.g., 20 centilogits). Inother embodiments, other combinations of scores can be compared, such asa comparison of segment A and segment B-A.

Further, in certain embodiments, the method can remove overlap wherethere is a large degree of overlap but not a proper subset.Specifically, the method can comprise the following steps: if the firstsegment respondents are a not a proper subset of the second segmentrespondents, then (a) if a first segment noteworthiness score is closeto a second segment noteworthiness score, then reducing thenoteworthiness ranking of the less actionable of the first segment andthe second segment; and (b) if the first segment noteworthiness score isnot close to the second segment noteworthiness score, then reducing thenoteworthiness ranking of the segment having a noteworthiness scorecloser to an average noteworthiness score; wherein the firstnoteworthiness score and the second noteworthiness score are close ifeither the scores differ by a predetermined percentage or amount; or adistance between the scores exceeds a difference between a benchmarkfirst percentile score and a benchmark second percentile score; andwherein the actionability of a segment is based on an ease or difficultyin addressing the response grouping.

For example, Executives and Salary Greater Than $100 k will haveoverlap, but neither is a perfect subset of the other. FIG. 9Billustrates such an overlap, where Executives is represented by circle Aand Salary Greater Than $100 k is represented by circle B, and theoverlap where A intersects be is represented by the portion labeled A∩B.In this example, if the scores for the overlapping segments are close,then the system treats the less actionable segment as less noteworthy.Determining whether scores are close can be based on an arbitrarynumber, but more typically, it will be the score change observed alongan agreed upon percentile difference that is meaningful to the company,for example, the distance in scores between the company in theirbenchmark at the 25th percentile and the company in their benchmark atthe 75th percentile. The actionability of a segment can based on an easeor difficulty in addressing the item. For example, Top Executives aremore actionable than Salary Greater than $100 k, because it is far lessawkward to get Top Executives in a room together to address an issuethan it is to call a meeting of people making more than $100 k, whichwould reveal salary information in a way that most companies choose toavoid.

If the overlapping segment scores are not close, then the system treatsthe closer-to-average segment as less noteworthy. For instance, ifExecutives are noteworthy and Salary Greater Than $100 k is similarlynoteworthy, and these are substantially the same set of people, then itis not helpful to state that both are noteworthy.

It is noted that the steps discussed herein for determiningnoteworthiness scores and adjusting noteworthiness rankings can becarried out in various orders. In one embodiment, the noteworthinessranking is first determined based on the composite score for eachresponse grouping; a change in the composite score for the responsegrouping from a previous composite score; and the actionability of theresponse grouping. Subsequently, adjustments to the noteworthinessrankings are made based on overlap. In other embodiments, adjustmentscan be carried out in different orders.

In certain embodiments, the displayed response groupings can include acombination of two or more of the respondent segments and the pluralityof survey items. Thus, instead of just looking at each of the segmentsand items, the system can also look at every combination of segments anditems together. For example, the system can look at Sales but also seeall of Sales' demographic groupings and all survey items responded to bySales employees. For example, if it is really Sales New Hires that arethe problem, the system can show this instead of focusing simply onSales being the problem. Or it can be that the survey item Appreciationin Sales is the problem. In one embodiment, for each segment showing upas noteworthy, the system decomposes the segment into all of its itemsand subsegments, adds them to the sort order, and runs the process againto see if the proper subset overlap process finds something morenoteworthy than just looking at the top-level segment.

In certain embodiments, sequenced demographics can be combined. Asequenced demographic is an ordinal segmenting. For example, one caneasily order salary bands, tenure bands, or commute lengths according toa sequence (e.g., lowest to highest, or shortest to longest). Anon-sequenced demographic might be job role, since one cannot easilysequence line cooks, wait staff, and maintenance personnel. The systemcan consider every possible set of two or more adjacent sequencedsegments to find those most interesting. For example, rather than showthat the Less Than 6 Months group and the 6 to 12 Month group scores areworth attention, it is more meaningful to say that the Less than 1 Yeargroup is the real problem. This can be done similarly for othersegments, such as salary ranges. The process for combining sequenceddemographics can be carried out, in one embodiment, by considering, inaddition to the demographics, combinations of sequenced demographics,and then performing the proper subset check discussed above. In such anembodiment, the system can add combinations of sequenced demographics tothe sort order, and run the process again to determine whether theproper subset overlap process discussed above finds a more noteworthycombination.

In certain embodiments, for non-sequenced demographics (such asdepartments and locations), combinations can be created by puttingtogether all the combinations that exclude exactly one segment. Forexample, a Sales department may have four teams: East, West, North, andSouth. In addition to looking at Sales overall and the individual teams,the system can combine together all but one, namely, (1) Sales exceptEast (West, North, South), (2) Sales except West (East, North, South),(3) Sales except North (East, West, South), and (4) Sales except South(East, West, North). If South is responding much more positively thanthe other teams, it is more interesting to identify that Sales exceptSouth is noteworthy than to simply identify Sales as noteworthy, andthus Sales except South is what should be presented to leadership asneeding focused attention. In another example, it is more interesting toidentify that “All the locations except headquarters” score poorly.Similar to combining sequenced demographics, this process can includeadding the above combinations to the sort order and then determiningwhether the proper subset overlap process discussed above finds a morenoteworthy combination.

In certain embodiments, the system can ensure that a celebration (anindication of a noteworthy positive ranking) does not have badcomponents. Thus, the noteworthiness ranking of a response grouping canbe based in part on whether an aspect of the response grouping fallsbelow or above a predetermined threshold. For example, each of thecomponents of an algorithm determining noteworthiness can have athreshold of “badness” that, if met, will automatically cause a segmentto be treated as substantially less noteworthy so that the segmenthaving some badness (e.g., a score below benchmark, or a large drop froma previous score), does not show up as a celebration.

Noteworthiness can also be based on a multiplier. For example, there canbe a multiplier per individual respondent, per segment, and/or per item.These multipliers can be based on any of the factors discussed herein,or any combination thereof. A segment multiplier, for example, can bebased on the average or aggregate salary of those associated with thesegment. Further, a segment multiplier can be based on how actionablethe segment is. Further, a segment multiplier can be based on how manytop performers are in a segment, where segments with more or a higherpercentage of top performers are given more weight.

An individual multiplier can be based on salary, but other bases for themultiplier can be used, such as respondent tenure, aggregate salary,respondent salary, respondent management level, respondent role,respondent performance ratings, or some combination thereof.

A survey item multiplier can be calculated by determining how thelogit-based item composite scores correlate to a business metric ofinterest. An example of one such method would be to use Pearson'sproduct-moment correlation coefficient to determine the extent to whichrespondent responses to various survey items relate to their individualproductivity. In other embodiments, the multipliers discussed herein canbe chosen simply based on a value the organization decides to give to agiven person, segment, or item.

Method for Generating Display

FIG. 7 is a flowchart for one method 150 for generating a displaysummarizing survey responses. In this embodiment, a system providessurvey items to survey respondents (operation 151). The system thenreceives survey responses thereto (operation 152). The system thendetermines a calibrated score for each response (operation 153). Thesystem then determines composite scores for each response grouping(e.g., each survey item and each segment) (operation 154). The systemthen provides a display based on the composite scores (operation 155).The invention is not limited to this exemplified method, as other oradditional operations can be included. For example, after determiningcomposite scores, the method can use one or more of the factorsdiscussed herein to determine a noteworthiness ranking for each responsegrouping, and the displayed response groupings can be ordered accordingto the determined noteworthiness rankings.

Alternative Display

FIG. 8 is a display 200 of a summary of survey responses according toanother embodiment. This graph is similar to the display of FIG. 3 inthat the position of the images 208 with respect to the first axis 202is based on a composite score. Further, like the display of FIG. 3, thesize of the image is based on one or more details about the associatedresponse grouping (e.g., total number of the survey respondents, totalnumber of possible survey respondents, or a multiplier). Other featuresof FIG. 3, such as coalescence and polarization indicators, can also beincluded in display 200.

The primary differences from FIG. 3 are that there are no arrowsindicating the change in composite score, and there is no list ofresponse groupings along a first axis. Rather than arrows, FIG. 8includes a second axis 204. The images 208 are located with respect tothe second axis 204 based on the change in composite score since thelast survey. Further, rather than listing the response groupings along afirst axis, the identity of the response groupings can be omitted, orcan be displayed by various alternative means, such as by placing textnext to the image, or by having descriptive text appear when the imageis selected or hovered over using a mouse pointer. In the exemplifiedembodiment, the image 206 corresponds with an item of particularinterest to the organization generating the display, though theinventions are not so limited.

Results for any segment or item (or combination thereof) can bedisplayed using this approach to show how response groupings within aselected organization compare to each other. Thus, for example,different departments within a company can each have a departmentorganizational score, the department organizational score being based onthe calibrated scores for each survey response of each respondent of therelevant department. Each department can be represented by an image(e.g., circle) on the graph corresponding with the departmentorganizational score.

Comparing Organization Composite Scores

The graph of FIG. 8 can also be used to display 200 a summary of surveyresponses for different organizations according to another embodiment.By this alternative understanding of FIG. 8, the display 200 includes afirst axis 202 corresponding with an organization composite score, theorganization composite score being based on the calibrated scores foreach survey response of each respondent of the organization.

The organization being surveyed has an image 206 corresponding with theorganization composite score for that organization. The display furtherincludes additional images 208 corresponding with the organizationalcomposite scores for other organizations. Each image is located withrespect to a first axis based on the associated organization compositescore. This enables the surveyed organization to see visually how itcompares to other organizations.

As shown, the exemplified display 200 can also include a second axis204. Each image can be located with respect to the second axis based ona change to the associated organization composite score. In otherembodiments, this feature can be omitted.

Advantages

The above methods avoid the aforementioned shortcomings of other methodsof summarizing survey responses, such as information loss, lack ofcalibration, the assumption that response options are equidistant fromone another, and the assumption that it is valid to calculate a standarddeviation from Likert-type scale response data. The methods describedherein can more appropriately weight passion and outliers, leading tobetter group decision-making. The technologic modifications solve theabove problems and improve the functioning of survey systems by, amongother things, helping organizations identify the most noteworthy resultsof a survey. This allows an organization to quickly and accuratelyidentify the areas most ripe for impactful change, thus allowing theorganization to focus its limited time and resources on those areas. Themethods also enable the comparison of survey items with other responsegroupings, such as categories of employees. The methods described hereinprovide an improvement to the capability of survey systems as a whole.

As used throughout, ranges are used as shorthand for describing each andevery value that is within the range. Any value within the range can beselected as the terminus of the range. In addition, all references citedherein are hereby incorporated by referenced in their entireties. In theevent of a conflict in a definition in the present disclosure and thatof a cited reference, the present disclosure controls.

While the invention or inventions have been described with respect tospecific examples, those skilled in the art will appreciate that thereare numerous variations and permutations of the above describedinvention(s). It is to be understood that other embodiments may beutilized and structural and functional modifications may be made withoutdeparting from the scope of the present invention(s). Thus, the spiritand scope should be construed broadly as set forth in the appendedclaims.

What is claimed is:
 1. A method of displaying survey results, the methodcomprising: a) for each of a plurality of survey items, receiving surveyresponses from survey respondents, wherein: i) each response is chosenfrom response options, the response options corresponding to an ordinalscale; ii) each respondent is associated with a collection of respondentsegments; and iii) each response to each item by each survey respondentcorresponds with a response score; b) for each item, determining acomposite score based on the response score for each response to theitem; c) for each respondent segment, determining a composite scorebased on the response score for each response associated with therespondent segment; and d) providing a display of response groupings,the displayed response groupings comprising at least a portion of theplurality of items and the collection of respondent segments; e) whereinthe response groupings are located on the display along a first axis;and f) wherein the display includes, for each response grouping, acorresponding image, the corresponding image being located with respectto the first axis or a distinct second axis according to anoteworthiness ranking, the noteworthiness ranking being based on atleast two of: i) the composite score for the response grouping; ii) achange in the composite score for the response grouping from a previouscomposite score for the response grouping; and iii) a sum of weights foreach respondent segment, where each survey respondent or possible surveyrespondent is assigned a weight.
 2. The method of claim 1 wherein thenoteworthiness ranking is based on each of: i) the composite score forthe response grouping; ii) a change in the composite score for theresponse grouping from a previous composite score for the responsegrouping; and iii) a sum of weights for each respondent segment, whereeach survey respondent or possible survey respondent is assigned aweight.
 3. The method of claim 1 wherein the response score for eachresponse is determined based on whether the response is positive ornegative, an integer assignment, or z-scores.
 4. The method of claim 1wherein the response score for each response is a calibrated score, thecalibrated score based on a probability that a person having thecollection of respondent segments associated with the survey respondentwould provide either: i) the response to the item or any of the responseoptions to the item that are lower on the ordinal scale; or ii) theresponse to the item or any of the response options to the item that arehigher on the ordinal scale.
 5. The method of claim 4 wherein thecalibrated scored is further based on benchmark data, and the benchmarkdata comprises a calibrated score for each response option for each itemfor each possible collection of respondent segments.
 6. The method ofclaim 4 wherein each calibrated score is a logit-based score.
 7. Themethod of claim 1, a) wherein the collection of respondent segmentscomprises demographic segments and department segments; and b) whereinthe demographic segments are associated with at least one of respondenttenure, respondent hours worked, respondent salary range, respondentmanagement level, and respondent role.
 8. A non-transitorycomputer-readable storage medium encoded with instructions which, whenexecuted on a processor, perform the method of: a) for each of aplurality of survey items, receiving survey responses from surveyrespondents, wherein: i) each response is chosen from response options,the response options corresponding to an ordinal scale; ii) eachrespondent is associated with a collection of respondent segments; andiii) each response to each item by each survey respondent correspondswith a response score; b) for each item, determining a composite scorebased on the response score for each response to the item; c) for eachrespondent segment, determining a composite score based on the responsescore for each response associated with the respondent segment; and d)providing a display of response groupings, the displayed responsegroupings comprising at least a portion of the plurality of items andthe collection of respondent segments; e) wherein the response groupingsare located on the display along a first axis; and f) wherein thedisplay includes, for each response grouping, a corresponding image, thecorresponding image being located with respect to the first axis or adistinct second axis according to a noteworthiness ranking, thenoteworthiness ranking being based on at least two of: i) the compositescore for the response grouping; ii) a change in the composite score forthe response grouping from a previous composite score for the responsegrouping; and iii) a sum of weights for each respondent segment, whereeach survey respondent or possible survey respondent is assigned aweight.
 9. The medium of claim 8 wherein the noteworthiness ranking isbased on each of: i) the composite score for the response grouping; ii)a change in the composite score for the response grouping from aprevious composite score for the response grouping; and iii) a sum ofweights for each respondent segment, where each survey respondent orpossible survey respondent is assigned a weight.
 10. The medium of claim8 wherein the response score for each response is determined based onwhether the response is positive or negative, an integer assignment, orz-scores.
 11. The medium of claim 8 wherein the response score for eachresponse is a calibrated score, the calibrated score based on aprobability that a person having the collection of respondent segmentsassociated with the survey respondent would provide either: i) theresponse to the item or any of the response options to the item that arelower on the ordinal scale; or ii) the response to the item or any ofthe response options to the item that are higher on the ordinal scale.12. The medium of claim 11 wherein the calibrated scored is furtherbased on benchmark data, and the benchmark data comprises a calibratedscore for each response option for each item for each possiblecollection of respondent segments.
 13. The method of claim 11 whereineach calibrated score is a logit-based score.
 14. The method of claim 1,a) wherein the collection of respondent segments comprises demographicsegments and department segments; and b) wherein the demographicsegments are associated with at least one of respondent tenure,respondent hours worked, respondent salary range, respondent managementlevel, and respondent role.
 15. A system for displaying survey results,the system comprising: a) respondent devices configured to receivesurvey items and communicate survey responses to the items; and b) aserver configured to carry out the steps of: i) for each item, receivingsurvey responses from respondent devices, wherein: (1) each response ischosen from response options, the response options corresponding to anordinal scale; (2) each respondent is associated with a collection ofrespondent segments; and (3) each response to each item by each surveyrespondent corresponds with a response score; ii) for each item,determining a composite score based on the response score for eachresponse to the item; iii) for each respondent segment, determining acomposite score based on the response score for each response associatedwith the respondent segment; and iv) providing a display of responsegroupings, the displayed response groupings comprising at least aportion of the plurality of items and the collection of respondentsegments; v) wherein the response groupings are located on the displayalong a first axis; and vi) wherein the display includes, for eachresponse grouping, a corresponding image, the corresponding image beinglocated with respect to the first axis or a distinct second axisaccording to a noteworthiness ranking, the noteworthiness ranking beingbased on at least two of: (1) the composite score for the responsegrouping; (2) a change in the composite score for the response groupingfrom a previous composite score for the response grouping; and (3) a sumof weights for each respondent segment, where each survey respondent orpossible survey respondent is assigned a weight.
 16. The system of claim15 wherein the noteworthiness ranking is based on each of: i) thecomposite score for the response grouping; ii) a change in the compositescore for the response grouping from a previous composite score for theresponse grouping; and iii) a sum of weights for each respondentsegment, where each survey respondent or possible survey respondent isassigned a weight.
 17. The system of claim 15 wherein the response scorefor each response is determined based on whether the response ispositive or negative, an integer assignment, or z-scores.
 18. The systemof claim 15 wherein the response score for each response is a calibratedscore, the calibrated score based on a probability that a person havingthe collection of respondent segments associated with the surveyrespondent would provide either: i) the response to the item or any ofthe response options to the item that are lower on the ordinal scale; orii) the response to the item or any of the response options to the itemthat are higher on the ordinal scale.
 19. The system of claim 18 whereinthe calibrated scored is further based on benchmark data, and thebenchmark data comprises a calibrated score for each response option foreach item for each possible collection of respondent segments.
 20. Thesystem of claim 18 wherein each calibrated score is a logit-based score.